Software developers often use tests to validate the functionality of their source code when changes have been made to the codebase and prior to deploying the code to production. If the tests pass, developers may assume that their code is working correctly and if the tests fail, developers may determine that the source code has bugs or other problems that need to be fixed. However, some tests may be non-deterministic, meaning that there are times when the tests pass and times when the tests fail without any changes to the source code.
Non-deterministic tests may have several causes. For instance, some tests are written based on wrong assumptions about their behaviors such as assuming fixed execution time for a certain task. Other tests have unexpected resource sharing with other programs, such as network, file system and memory. Additionally, some tests contain concurrency bugs themselves. Non-determinism in software tests is a major impediment to effectively improving and evolving a codebase, especially as the codebase grows very large. These non-deterministic, flaky, tests can cause multiple problems including providing false success results when new code has a subtle non-deterministic bug or causing flaky libraries' client applications to be unpredictable. Due to flakiness, it is difficult to reproduce the incorrect behavior and pinpoint the behavior's cause within source code. Therefore, as recognized by the inventors, there should be a tool to help developers effectively and efficiently determine the root cause of non-deterministic tests.